{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# The Image Classification Dataset\n",
    ":label:`sec_fashion_mnist`\n",
    "\n",
    "In :numref:`sec_naive_bayes`, we trained a naive Bayes classifier,\n",
    "using the MNIST dataset introduced in 1998 :cite:`LeCun.Bottou.Bengio.ea.1998`. \n",
    "While MNIST had a good run as a benchmark dataset, \n",
    "even simple models by today's standards achieve classification accuracy over 95%\n",
    "making it unsuitable for distinguishing between stronger models and weaker ones. \n",
    "Today, MNIST serves as more of sanity checks than as a benchmark.\n",
    "To up the ante just a bit, we will focus our discussion in the coming sections\n",
    "on the qualitatively similar, but comparatively complex Fashion-MNIST \n",
    "dataset :cite:`Xiao.Rasul.Vollgraf.2017`, which was released in 2017."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "attributes": {
     "classes": [],
     "id": "",
     "n": "7"
    }
   },
   "outputs": [],
   "source": [
    "%load ../utils/djl-imports\n",
    "%load ../utils/StopWatch.java"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import ai.djl.basicdataset.cv.classification.*;\n",
    "import java.awt.image.BufferedImage;\n",
    "import java.awt.Graphics2D;\n",
    "import java.awt.Graphics;\n",
    "import java.awt.Color;\n",
    "import java.awt.Dimension;\n",
    "import java.awt.FlowLayout;\n",
    "import java.awt.Component;\n",
    "import javax.swing.*;"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Getting the Dataset\n",
    "\n",
    "Just as with MNIST, DJL makes it easy to download and load the Fashion-MNIST dataset into memory via the `FashionMnist` class contained in `ai.djl.basicdataset`.\n",
    "We briefly work through the mechanics of loading and exploring the dataset below. \n",
    "Please refer to :numref:`sec_naive_bayes` for more details on loading data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us first define the `getDataset()` function \n",
    "that obtains and reads the Fashion-MNIST dataset. \n",
    "It returns the dataset for the training set or the validation set depending\n",
    "on the passed in `usage` (`Dataset.Usage.TRAIN` for training and `Dataset.Usage.TEST` for validation). \n",
    "You can then call `getData(manager)` on the dataset to get the corresponding iterator.\n",
    "It also takes in the `batchSize` and `randomShuffle` which dictates\n",
    "the size of each batch and whether to randomly shuffle the data respectively."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "int batchSize = 256;\n",
    "boolean randomShuffle = true;\n",
    "\n",
    "FashionMnist mnistTrain = FashionMnist.builder()\n",
    "        .optUsage(Dataset.Usage.TRAIN)\n",
    "        .setSampling(batchSize, randomShuffle)\n",
    "        .optLimit(Long.getLong(\"DATASET_LIMIT\", Long.MAX_VALUE))\n",
    "        .build();\n",
    "\n",
    "FashionMnist mnistTest = FashionMnist.builder()\n",
    "        .optUsage(Dataset.Usage.TEST)\n",
    "        .setSampling(batchSize, randomShuffle)\n",
    "        .optLimit(Long.getLong(\"DATASET_LIMIT\", Long.MAX_VALUE))\n",
    "        .build();\n",
    "\n",
    "mnistTrain.prepare();\n",
    "mnistTest.prepare();\n",
    "\n",
    "NDManager manager = NDManager.newBaseManager();"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Fashion-MNIST consists of images from 10 categories, each represented \n",
    "by 60k images in the training set and by 10k in the test set. \n",
    "Consequently the training set and the test set \n",
    "contain 60k and 10k images, respectively."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "System.out.println(mnistTrain.size());\n",
    "System.out.println(mnistTest.size());"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The images in Fashion-MNIST are associated with the following categories: \n",
    "t-shirt, trousers, pullover, dress, coat, sandal, shirt, sneaker, bag and ankle boot. \n",
    "The following function converts between numeric label indices and their names in text."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "attributes": {
     "classes": [],
     "id": "",
     "n": "25"
    }
   },
   "outputs": [],
   "source": [
    "// Saved in the FashionMnist class for later use\n",
    "public String[] getFashionMnistLabels(int[] labelIndices) {\n",
    "    String[] textLabels = {\"t-shirt\", \"trouser\", \"pullover\", \"dress\", \"coat\",\n",
    "                   \"sandal\", \"shirt\", \"sneaker\", \"bag\", \"ankle boot\"};\n",
    "    String[] convertedLabels = new String[labelIndices.length];\n",
    "    for (int i = 0; i < labelIndices.length; i++) {\n",
    "        convertedLabels[i] = textLabels[labelIndices[i]];\n",
    "    }\n",
    "    return convertedLabels;\n",
    "}\n",
    "\n",
    "public String getFashionMnistLabel(int labelIndice) {\n",
    "    String[] textLabels = {\"t-shirt\", \"trouser\", \"pullover\", \"dress\", \"coat\",\n",
    "                   \"sandal\", \"shirt\", \"sneaker\", \"bag\", \"ankle boot\"};\n",
    "    return textLabels[labelIndice];\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now create a function to visualize these examples.\n",
    "Don't worry too much about the specifics of visualization.\n",
    "This is simply just to help intuitively understand the data.\n",
    "We essentially read in a number of datapoints and convert their\n",
    "RGB value from 0-255 to between 0-1. We then set the color as grayscale\n",
    "and then display it along with their labels in an external window."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "public class ImagePanel extends JPanel {\n",
    "    int SCALE;\n",
    "    BufferedImage img;\n",
    "    \n",
    "    public ImagePanel() {\n",
    "        this.SCALE = 1;\n",
    "    }\n",
    "    public ImagePanel(int scale, BufferedImage img) {\n",
    "        this.SCALE = scale;\n",
    "        this.img = img;\n",
    "    }\n",
    "    @Override\n",
    "    protected void paintComponent(Graphics g) {\n",
    "        Graphics2D g2d = (Graphics2D)g;\n",
    "        g2d.scale(SCALE, SCALE);\n",
    "        g2d.drawImage(this.img, 0, 0, this);\n",
    "    }\n",
    "}\n",
    "\n",
    "public class Container extends JPanel {\n",
    "    public Container(String label) {\n",
    "        setLayout(new BoxLayout(this, BoxLayout.Y_AXIS));\n",
    "        JLabel l = new JLabel(label, JLabel.CENTER);\n",
    "        l.setAlignmentX(Component.CENTER_ALIGNMENT);\n",
    "        add(l);\n",
    "    }\n",
    "    public Container(String trueLabel, String predLabel) {\n",
    "        setLayout(new BoxLayout(this, BoxLayout.Y_AXIS));\n",
    "        JLabel l = new JLabel(trueLabel, JLabel.CENTER);\n",
    "        l.setAlignmentX(Component.CENTER_ALIGNMENT);\n",
    "        add(l);\n",
    "        JLabel l2 = new JLabel(predLabel, JLabel.CENTER);\n",
    "        l2.setAlignmentX(Component.CENTER_ALIGNMENT);\n",
    "        add(l2);\n",
    "    }\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "// Saved in the FashionMnistUtils class for later use\n",
    "public void showImages(ArrayDataset dataset, \n",
    "                       int number, int WIDTH, int HEIGHT, int SCALE, \n",
    "                       NDManager manager) \n",
    "    throws IOException, TranslateException {\n",
    "    // Plot a list of images\n",
    "    JFrame frame = new JFrame(\"Fashion Mnist\");\n",
    "    for (int record = 0; record < number; record++) {\n",
    "        NDArray X = dataset.get(manager, record).getData().get(0).squeeze(-1);\n",
    "        int y = (int)dataset.get(manager, record).getLabels().get(0).getFloat();\n",
    "        BufferedImage img = new BufferedImage(WIDTH, HEIGHT, BufferedImage.TYPE_BYTE_GRAY);\n",
    "        Graphics2D g = (Graphics2D) img.getGraphics();\n",
    "        for(int i = 0; i < WIDTH; i++) {\n",
    "            for(int j = 0; j < HEIGHT; j++) {\n",
    "                float c = X.getFloat(j, i) / 255;  // scale down to between 0 and 1\n",
    "                g.setColor(new Color(c, c, c)); // set as a gray color\n",
    "                g.fillRect(i, j, 1, 1);\n",
    "            }\n",
    "        }\n",
    "        JPanel panel = new ImagePanel(SCALE, img);\n",
    "        panel.setPreferredSize(new Dimension(WIDTH * SCALE, HEIGHT * SCALE));\n",
    "        JPanel container = new Container(getFashionMnistLabel(y));\n",
    "        container.add(panel);\n",
    "        frame.getContentPane().add(container);\n",
    "    }\n",
    "    frame.getContentPane().setLayout(new FlowLayout());\n",
    "    frame.pack();\n",
    "    frame.setVisible(true);\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here are the images and their corresponding labels (in text)\n",
    "for the first few examples in the training dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "final int SCALE = 4;\n",
    "final int WIDTH = 28;\n",
    "final int HEIGHT = 28;\n",
    "\n",
    "/* Uncomment the following line and run to display images.\n",
    "   It will open in another window. */\n",
    "// showImages(mnistTrain, 18, WIDTH, HEIGHT, SCALE, manager);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Fashion Mnist labels.](https://d2l-java-resources.s3.amazonaws.com/img/fashion_mnist_labels.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Reading a Minibatch\n",
    "\n",
    "To make our life easier when reading from the training and test sets,\n",
    "we use the `getData(manager)`.\n",
    "Recall that at each iteration, `getData(manager)` \n",
    "reads a minibatch of data with size `batchSize` each time.\n",
    "We then get the X and y by calling `getData()` and `getLabels()` on each batch respectively.\n",
    "\n",
    "Note: During training, reading data can be a significant performance bottleneck, \n",
    "especially when our model is simple or when our computer is fast. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us look at the time it takes to read the training data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "StopWatch stopWatch = new StopWatch();\n",
    "stopWatch.start();\n",
    "for (Batch batch : mnistTrain.getData(manager)) {\n",
    "    NDArray x = batch.getData().head();\n",
    "    NDArray y = batch.getLabels().head();\n",
    "}\n",
    "System.out.println(String.format(\"%.2f sec\", stopWatch.stop()));"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are now ready to work with the Fashion-MNIST dataset in the sections that follow.\n",
    "\n",
    "## Summary\n",
    "\n",
    "* Fashion-MNIST is an apparel classification dataset consisting of images representing 10 categories. \n",
    " * We will use this dataset in subsequent sections and chapters to evaluate various classification algorithms.\n",
    "* We store the shape of each image with height $h$ width $w$ pixels as $h \\times w$ or `(h, w)`.\n",
    "* Data iterators are a key component for efficient performance. Rely on well-implemented iterators that exploit multi-threading to avoid slowing down your training loop.\n",
    "\n",
    "## Exercises\n",
    "\n",
    "1. Does reducing the `batchSize` (for instance, to 1) affect read performance?\n",
    "1. Use the DJL documentation to see which other datasets are available in `ai.djl.basicdataset`."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Java",
   "language": "java",
   "name": "java"
  },
  "language_info": {
   "codemirror_mode": "java",
   "file_extension": ".jshell",
   "mimetype": "text/x-java-source",
   "name": "Java",
   "pygments_lexer": "java",
   "version": "14.0.2+12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
